613928 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.SolrTestCaseJ4 ###Starting testKillPullReplica 613963 INFO (OverseerThreadFactory-7998-thread-1) [n: c:pull_replica_test_kill_pull_replica s: r: x: t:] o.a.s.c.a.c.CreateCollectionCmd Create collection pull_replica_test_kill_pull_replica 614076 INFO (OverseerThreadFactory-7998-thread-1) [n: c:pull_replica_test_kill_pull_replica s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { "core":"pull_replica_test_kill_pull_replica_shard1_replica_n1", "node_name":"127.0.0.1:35647_solr", "base_url":"https://127.0.0.1:35647/solr", "collection":"pull_replica_test_kill_pull_replica", "shard":"shard1", "state":"down", "type":"NRT", "operation":"ADDREPLICA", "waitForFinalState":"false"} 614077 INFO (OverseerThreadFactory-7998-thread-1) [n: c:pull_replica_test_kill_pull_replica s: r: x: t:] o.a.s.c.o.SliceMutator createReplica() { "core":"pull_replica_test_kill_pull_replica_shard1_replica_p2", "node_name":"127.0.0.1:38207_solr", "base_url":"https://127.0.0.1:38207/solr", "collection":"pull_replica_test_kill_pull_replica", "shard":"shard1", "state":"down", "type":"PULL", "operation":"ADDREPLICA", "waitForFinalState":"false"} 614114 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&collection=pull_replica_test_kill_pull_replica&version=2&replicaType=PULL&coreNodeName=core_node4&name=pull_replica_test_kill_pull_replica_shard1_replica_p2&action=CREATE&numShards=1&shard=shard1&wt=javabin 614116 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&collection=pull_replica_test_kill_pull_replica&version=2&replicaType=NRT&coreNodeName=core_node3&name=pull_replica_test_kill_pull_replica_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&wt=javabin 614126 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.11.1 614128 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.11.1 614129 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.s.IndexSchema Schema name=minimal 614130 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.s.IndexSchema Loaded schema minimal/1.7 with uniqueid field id 614130 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_p2' using configuration from configset conf, trusted=true 614130 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.s.IndexSchema Schema name=minimal 614130 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.s.IndexSchema Loaded schema minimal/1.7 with uniqueid field id 614130 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_n1' using configuration from configset conf, trusted=true 614131 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.SolrCore Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node2/pull_replica_test_kill_pull_replica_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node2/pull_replica_test_kill_pull_replica_shard1_replica_n1/data/] 614131 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.SolrCore Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node1/pull_replica_test_kill_pull_replica_shard1_replica_p2], dataDir=[/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node1/pull_replica_test_kill_pull_replica_shard1_replica_p2/data/] 614157 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 614161 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.u.CommitTracker Hard AutoCommit: disabled 614161 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.u.CommitTracker Soft AutoCommit: disabled 614167 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 614167 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 614167 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 614169 INFO (searcherExecutor-8014-thread-1-processing-pull_replica_test_kill_pull_replica_shard1_replica_p2 null-11280 core_node4 127.0.0.1:38207_solr pull_replica_test_kill_pull_replica shard1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 614171 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 614175 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog 614175 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 614175 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.u.CommitTracker Hard AutoCommit: disabled 614175 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.u.CommitTracker Soft AutoCommit: disabled 614177 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 614177 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 614177 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 614177 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.ZkController pull_replica_test_kill_pull_replica_shard1_replica_p2 starting background replication from leader 614177 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:01 614178 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms 614178 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 614179 INFO (searcherExecutor-8016-thread-1-processing-pull_replica_test_kill_pull_replica_shard1_replica_n1 null-11280 core_node3 127.0.0.1:35647_solr pull_replica_test_kill_pull_replica shard1) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 614182 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ZkShardTerms Successful update of terms at /collections/pull_replica_test_kill_pull_replica/terms/shard1 to Terms{values={core_node3=0}, version=0} for registerTerm 614182 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/pull_replica_test_kill_pull_replica/leaders/shard1 614189 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. 614189 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync 614189 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/ 614190 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me 614190 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.SyncStrategy https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/ has no replicas 614190 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/pull_replica_test_kill_pull_replica/leaders/shard1/leader after winning as /collections/pull_replica_test_kill_pull_replica/leader_elect/shard1/election/72201048094015494-core_node3-n_0000000000 614193 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/ shard1 614195 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11280] o.a.s.c.ZkController I am the leader, no recovery necessary 614200 INFO (qtp1402283490-12961-null-11280) [n:127.0.0.1:35647_solr c: s: r: x: t:null-11280] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_n1&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=85 614816 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.ReplicationHandler Polling for index modifications 614817 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Updated leaderUrl to https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/ 614842 INFO (qtp1402283490-12967-null-11281) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11281] o.a.s.c.S.Request webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 614842 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Leader's generation: 1 614842 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Leader's version: 0 614842 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Follower's generation: 1 614842 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Follower's version: 0 614842 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Nothing to replicate, leader's version is 0 615181 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 198] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 615184 INFO (qtp1483242594-12956-null-11280) [n:127.0.0.1:38207_solr c: s: r: x: t:null-11280] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=pull_replica_test_kill_pull_replica_shard1_replica_p2&action=CREATE&numShards=1&collection=pull_replica_test_kill_pull_replica&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=1070 615186 INFO (qtp1483242594-12962-null-11279) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s: r: x: t:null-11279] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas 615186 INFO (qtp1483242594-12962-null-11279) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s: r: x: t:null-11279] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=1&collection.configName=conf&name=pull_replica_test_kill_pull_replica&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=1231 615187 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.SolrCloudTestCase waitForState (pull_replica_test_kill_pull_replica): Expected collection to be created with 1 shard and 2 replicas 615187 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.SolrCloudTestCase active replica count: 2 expected replica count: 2 615210 INFO (qtp1402283490-12961-null-11282) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11282] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11282 hits=0 status=0 QTime=0 615210 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node3 (https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/) has all 0 docs 615216 INFO (qtp1483242594-12966-null-11283) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11283] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11283 hits=0 status=0 QTime=0 615217 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node4 (https://127.0.0.1:38207/solr/pull_replica_test_kill_pull_replica_shard1_replica_p2/) has all 0 docs 615225 INFO (qtp1402283490-12961-null-11284) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11284] o.a.s.c.ZkShardTerms Successful update of terms at /collections/pull_replica_test_kill_pull_replica/terms/shard1 to Terms{values={core_node3=1}, version=1} for ensureHighestTermsAreNotZero 615225 INFO (qtp1402283490-12961-null-11284) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11284] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1812808767375409152)]} 0 3 615230 INFO (searcherExecutor-8016-thread-1-processing-pull_replica_test_kill_pull_replica_shard1_replica_n1 null-11285 core_node3 127.0.0.1:35647_solr pull_replica_test_kill_pull_replica shard1) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11285] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 615230 INFO (qtp1402283490-12967-null-11285) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11285] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={_stateVer_=pull_replica_test_kill_pull_replica:6&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 3 615236 INFO (qtp1402283490-12963-null-11286) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11286] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11286 hits=1 status=0 QTime=0 615237 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node3 (https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/) has all 1 docs 615244 INFO (qtp1483242594-12962-null-11287) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11287] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11287 hits=0 status=0 QTime=0 615445 INFO (qtp1483242594-12958-null-11288) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11288] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11288 hits=0 status=0 QTime=0 615647 INFO (qtp1483242594-12960-null-11289) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11289] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11289 hits=0 status=0 QTime=0 615843 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.ReplicationHandler Polling for index modifications 615843 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher leaderUrl didn't change 615844 INFO (qtp1402283490-12957-null-11290) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11290] o.a.s.c.S.Request webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 615844 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Leader's generation: 2 615844 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Leader's version: 1728829162005 615844 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Follower's generation: 1 615844 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Follower's version: 0 615844 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Starting replication process 615846 INFO (qtp1402283490-12959-null-11291) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11291] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=0 615846 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Number of files in latest index in leader: 15 615847 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(ByteBuffersDirectory@4e5f711c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e5566f7) 615847 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Download files to dir: [segments_1] 615847 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher tmpIndexDir_type : class org.apache.lucene.tests.store.MockDirectoryWrapper , ByteBuffersDirectory@4e5f711c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e5566f7 615847 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.kdm size=143 checksum=4092127899 alwaysDownload=true 615848 INFO (qtp1402283490-12965-null-11292) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11292] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.kdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615848 INFO (qtp1483242594-12964-null-11293) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11293] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11293 hits=0 status=0 QTime=0 615849 DEBUG (qtp1402283490-12965-null-11292) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11292] o.a.s.h.ReplicationHandler Wrote 143 bytes for file _0.kdm 615850 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 143 bytes of file: _0.kdm 615850 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.si size=473 checksum=1097052905 alwaysDownload=true 615851 INFO (qtp1402283490-12963-null-11294) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11294] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615851 DEBUG (qtp1402283490-12963-null-11294) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11294] o.a.s.h.ReplicationHandler Wrote 473 bytes for file _0.si 615851 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 473 bytes of file: _0.si 615851 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.kdi size=68 checksum=363169705 alwaysDownload=true 615852 INFO (qtp1402283490-12961-null-11295) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11295] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.kdi&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615852 DEBUG (qtp1402283490-12961-null-11295) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11295] o.a.s.h.ReplicationHandler Wrote 68 bytes for file _0.kdi 615853 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 68 bytes of file: _0.kdi 615853 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.fdm size=157 checksum=841077555 alwaysDownload=true 615854 INFO (qtp1402283490-12967-null-11296) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11296] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615854 DEBUG (qtp1402283490-12967-null-11296) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11296] o.a.s.h.ReplicationHandler Wrote 157 bytes for file _0.fdm 615855 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 157 bytes of file: _0.fdm 615855 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.doc size=77 checksum=985266366 alwaysDownload=true 615855 INFO (qtp1402283490-12957-null-11297) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11297] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Lucene99_0.doc&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615856 DEBUG (qtp1402283490-12957-null-11297) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11297] o.a.s.h.ReplicationHandler Wrote 77 bytes for file _0_Lucene99_0.doc 615856 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 77 bytes of file: _0_Lucene99_0.doc 615856 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.tim size=102 checksum=1906253852 alwaysDownload=true 615857 INFO (qtp1402283490-12959-null-11298) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11298] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Lucene99_0.tim&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615857 DEBUG (qtp1402283490-12959-null-11298) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11298] o.a.s.h.ReplicationHandler Wrote 102 bytes for file _0_Lucene99_0.tim 615857 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 102 bytes of file: _0_Lucene99_0.tim 615858 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.fdx size=64 checksum=4008489842 alwaysDownload=true 615858 INFO (qtp1402283490-12965-null-11299) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11299] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615858 DEBUG (qtp1402283490-12965-null-11299) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11299] o.a.s.h.ReplicationHandler Wrote 64 bytes for file _0.fdx 615859 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 64 bytes of file: _0.fdx 615859 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.tip size=74 checksum=204394137 alwaysDownload=true 615859 INFO (qtp1402283490-12963-null-11300) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11300] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Lucene99_0.tip&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615860 DEBUG (qtp1402283490-12963-null-11300) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11300] o.a.s.h.ReplicationHandler Wrote 74 bytes for file _0_Lucene99_0.tip 615860 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 74 bytes of file: _0_Lucene99_0.tip 615860 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.fdt size=114 checksum=2080799656 alwaysDownload=true 615862 INFO (qtp1402283490-12961-null-11301) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11301] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615863 DEBUG (qtp1402283490-12961-null-11301) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11301] o.a.s.h.ReplicationHandler Wrote 114 bytes for file _0.fdt 615863 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 114 bytes of file: _0.fdt 615863 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0_Lucene90_0.dvm size=758 checksum=3872417709 alwaysDownload=true 615864 INFO (qtp1402283490-12967-null-11302) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11302] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Lucene90_0.dvm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615864 DEBUG (qtp1402283490-12967-null-11302) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11302] o.a.s.h.ReplicationHandler Wrote 758 bytes for file _0_Lucene90_0.dvm 615865 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 758 bytes of file: _0_Lucene90_0.dvm 615865 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0_Lucene90_0.dvd size=81 checksum=3626414175 alwaysDownload=true 615865 INFO (qtp1402283490-12957-null-11303) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11303] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Lucene90_0.dvd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615891 DEBUG (qtp1402283490-12957-null-11303) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11303] o.a.s.h.ReplicationHandler Wrote 81 bytes for file _0_Lucene90_0.dvd 615891 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 81 bytes of file: _0_Lucene90_0.dvd 615892 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.kdd size=79 checksum=3053912650 alwaysDownload=true 615892 INFO (qtp1402283490-12959-null-11304) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11304] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.kdd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615893 DEBUG (qtp1402283490-12959-null-11304) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11304] o.a.s.h.ReplicationHandler Wrote 79 bytes for file _0.kdd 615893 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 79 bytes of file: _0.kdd 615893 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.tmd size=252 checksum=95871406 alwaysDownload=true 615894 INFO (qtp1402283490-12965-null-11305) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11305] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Lucene99_0.tmd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615894 DEBUG (qtp1402283490-12965-null-11305) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11305] o.a.s.h.ReplicationHandler Wrote 252 bytes for file _0_Lucene99_0.tmd 615894 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 252 bytes of file: _0_Lucene99_0.tmd 615894 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=_0.fnm size=660 checksum=2938713832 alwaysDownload=true 615895 INFO (qtp1402283490-12963-null-11306) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11306] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615896 DEBUG (qtp1402283490-12963-null-11306) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11306] o.a.s.h.ReplicationHandler Wrote 660 bytes for file _0.fnm 615896 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 660 bytes of file: _0.fnm 615896 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Downloading file=segments_2 size=220 checksum=2210641625 alwaysDownload=true 615897 INFO (qtp1402283490-12961-null-11307) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11307] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 615897 DEBUG (qtp1402283490-12961-null-11307) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11307] o.a.s.h.ReplicationHandler Wrote 220 bytes for file segments_2 615897 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Fetched and wrote 220 bytes of file: segments_2 615897 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Bytes downloaded: 3322, Bytes skipped downloading: 0 615897 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=3322) : 0 secs (null bytes/sec) to MockDirectoryWrapper(ByteBuffersDirectory@4e5f711c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e5566f7) 615898 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher From dir files: [_0.fdm, _0.fdt, _0.fdx, _0.fnm, _0.kdd, _0.kdi, _0.kdm, _0.si, _0_Lucene90_0.dvd, _0_Lucene90_0.dvm, _0_Lucene99_0.doc, _0_Lucene99_0.tim, _0_Lucene99_0.tip, _0_Lucene99_0.tmd, segments_2] 615898 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher To dir files: [segments_1] 615898 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.kdm 615898 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.si 615898 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.kdi 615898 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.fdm 615898 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0_Lucene99_0.doc 615898 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0_Lucene99_0.tim 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.fdx 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0_Lucene99_0.tip 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.fdt 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0_Lucene90_0.dvm 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0_Lucene90_0.dvd 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.kdd 615899 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0_Lucene99_0.tmd 615900 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: _0.fnm 615900 DEBUG (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.h.IndexFetcher Moving file: segments_2 615900 INFO (indexFetcher-8025-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 615901 INFO (searcherExecutor-8014-thread-1-processing-pull_replica_test_kill_pull_replica_shard1_replica_p2 null-11280 core_node4 127.0.0.1:38207_solr pull_replica_test_kill_pull_replica shard1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11280] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 615963 INFO (OverseerCollectionConfigSetProcessor-72201048094015494-127.0.0.1:35647_solr-n_0000000000) [n: c: s: r: x: t:] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper 616050 INFO (qtp1483242594-12956-null-11308) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11308] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11308 hits=1 status=0 QTime=0 616050 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node4 (https://127.0.0.1:38207/solr/pull_replica_test_kill_pull_replica_shard1_replica_p2/) has all 1 docs 616051 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.s.Server Stopped Server@55bd860c{STOPPING}[10.0.22,sto=0] 616054 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.s.AbstractConnector Stopped ServerConnector@c0276d8{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:0} 616054 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.CoreContainer Shutting down CoreContainer instance=365621900 616054 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:38207_solr 616055 INFO (zkCallback-8007-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1) 616055 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1) 616055 INFO (zkCallback-7989-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1) 616055 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.ZkController Publish this node as DOWN... 616055 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.ZkController Publish node=127.0.0.1:38207_solr as DOWN 616056 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 206] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [1]) 616057 INFO (zkCallback-7989-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 206] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [1]) 616072 INFO (coreCloseExecutor-8029-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@60ffa711 pull_replica_test_kill_pull_replica_shard1_replica_p2 616072 INFO (coreCloseExecutor-8029-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2 tag=SolrCore@60ffa711 616072 INFO (coreCloseExecutor-8029-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.pull_replica_test_kill_pull_replica.shard1.leader tag=SolrCore@60ffa711 616073 INFO (coreCloseExecutor-8029-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 616074 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null 616074 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null 616074 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null 616074 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null 616186 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4d4bb218{/solr,file:///home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-cwd/,STOPPED} 616187 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.SolrCloudTestCase waitForState (pull_replica_test_kill_pull_replica): Replica not removed 616189 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.SolrCloudTestCase waitForState (pull_replica_test_kill_pull_replica): Didn't update state 616192 INFO (qtp1402283490-12957-null-11309) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11309] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={wt=javabin&version=2}{add=[2 (1812808768388333568)]} 0 1 616195 INFO (searcherExecutor-8016-thread-1-processing-pull_replica_test_kill_pull_replica_shard1_replica_n1 null-11310 core_node3 127.0.0.1:35647_solr pull_replica_test_kill_pull_replica shard1) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11310] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 616195 INFO (qtp1402283490-12959-null-11310) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11310] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={_stateVer_=pull_replica_test_kill_pull_replica:6&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2 616202 INFO (qtp1402283490-12967-null-11311) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11311] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11311 hits=2 status=0 QTime=0 616202 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node3 (https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/) has all 2 docs 616202 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=38207) 616204 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.e.JettySolrRunner Trying to start Jetty on port 38207 try number 2 ... 616204 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.s.Server jetty-10.0.22; built: 2024-06-27T16:03:51.502Z; git: 5c8471e852d377fd726ad9b1692c35ffc5febb09; jvm 11.0.24+8 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38207, zkHost=127.0.0.1:34941/solr} 616214 ERROR (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.s.CoreContainerProvider ___ _ Welcome to Apache Solr™ version 9.8.0 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.s.CoreContainerProvider / __| ___| |_ _ Starting in cloud mode on port null 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_| Install dir: /home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr 616214 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.s.CoreContainerProvider |___/\___/_|_| Start time: 2024-10-13T14:19:22.992951Z 616215 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.s.CoreContainerProvider Solr Home: /home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node1 (source: servlet config: solr.solr.home) 616215 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 616216 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper 616218 INFO (zkConnectionManagerCallback-8032-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 616218 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 616218 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes 616218 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading... 616220 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.SolrXmlConfig Metrics collection is disabled. 616323 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config Trusting all certificates configured for Client@389930c6[provider=null,keyStore=null,trustStore=null] 616323 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@389930c6[provider=null,keyStore=null,trustStore=null] 616327 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config Trusting all certificates configured for Client@3a930dc8[provider=null,keyStore=null,trustStore=null] 616327 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@3a930dc8[provider=null,keyStore=null,trustStore=null] 616328 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config Trusting all certificates configured for Client@3261b6fb[provider=null,keyStore=null,trustStore=null] 616328 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@3261b6fb[provider=null,keyStore=null,trustStore=null] 616329 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config Trusting all certificates configured for Client@4afb7584[provider=null,keyStore=null,trustStore=null] 616329 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@4afb7584[provider=null,keyStore=null,trustStore=null] 616330 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34941/solr 616330 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates. 616331 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper 616332 INFO (zkConnectionManagerCallback-8045-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ConnectionManager zkClient has connected 616332 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 616337 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html 616337 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 616339 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates. 616340 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.ZkController Publish node=127.0.0.1:38207_solr as DOWN 616342 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38207_solr 616343 INFO (zkCallback-8007-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 616343 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 616345 WARN (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.CoreContainer Not all security plugins configured! authentication=disabled authorization=disabled. Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external. See https://s.apache.org/solrsecurity for more info 616349 INFO (zkCallback-8044-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) 616363 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node1 616364 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.CorePropertiesLocator Cores are: [pull_replica_test_kill_pull_replica_shard1_replica_p2] 616380 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.11.1 616382 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.s.IndexSchema Schema name=minimal 616382 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.s.IndexSchema Loaded schema minimal/1.7 with uniqueid field id 616387 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.11.1 616388 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.s.IndexSchema Schema name=minimal 616389 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.s.IndexSchema Loaded schema minimal/1.7 with uniqueid field id 616389 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_kill_pull_replica_shard1_replica_p2' using configuration from configset conf, trusted=true 616389 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.SolrCore Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node1/pull_replica_test_kill_pull_replica_shard1_replica_p2], dataDir=[/home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-tmp/org.apache.solr.cloud.TestPullReplica_F30CC837FDD0DC28-001/tempDir-001/node1/pull_replica_test_kill_pull_replica_shard1_replica_p2/data/] 616401 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 616415 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3b50f83c{/solr,file:///home/jenkins/jenkins-slave/workspace/Solr/Solr-Check-9.x/solr/core/build/tmp/tests-cwd/,AVAILABLE} 616416 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.e.j.s.AbstractConnector Started ServerConnector@38bf722b{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:38207} 616416 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n:127.0.0.1:38207_solr c: s: r: x: t:] o.e.j.s.Server Started Server@5b0a9c54{STARTING}[10.0.22,sto=0] @616451ms 616419 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer 616422 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.u.CommitTracker Hard AutoCommit: disabled 616423 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.u.CommitTracker Soft AutoCommit: disabled 616424 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf 616425 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf 616425 INFO (coreLoadExecutor-8050-thread-1-processing-127.0.0.1:38207_solr) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 616426 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.SolrCloudTestCase waitForState (pull_replica_test_kill_pull_replica): Replica not added 616426 INFO (searcherExecutor-8053-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 616434 INFO (coreZkRegister-8033-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.ZkController pull_replica_test_kill_pull_replica_shard1_replica_p2 starting background replication from leader 616434 INFO (coreZkRegister-8033-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:01 616442 INFO (coreZkRegister-8033-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms 616442 INFO (coreZkRegister-8033-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 616443 INFO (coreZkRegister-8033-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.ZkController Core needs to recover:pull_replica_test_kill_pull_replica_shard1_replica_p2 616443 DEBUG (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Polling for index modifications 616444 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Updated leaderUrl to https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/ 616450 INFO (updateExecutor-8039-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.u.DefaultSolrCoreState Running recovery 616453 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true 616453 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Publishing state of core [pull_replica_test_kill_pull_replica_shard1_replica_p2] as recovering, leader is [https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/] and I am [https://127.0.0.1:38207/solr/pull_replica_test_kill_pull_replica_shard1_replica_p2/] 616455 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 217] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616455 INFO (zkCallback-8044-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 217] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616455 INFO (zkCallback-8007-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 217] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616455 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Starting Replication Recovery. 616456 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Stopping background replicate from leader process 616456 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.ZkController pull_replica_test_kill_pull_replica_shard1_replica_p2 stopping background replication from leader 616476 INFO (qtp1402283490-12957-null-11312) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11312] o.a.s.c.S.Request webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 616477 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Leader's generation: 3 616477 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Leader's version: 1728829162971 616477 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower's generation: 1 616477 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower's version: 0 616477 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Starting replication process 616478 INFO (qtp1402283490-12967-null-11313) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11313] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=0 616480 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Number of files in latest index in leader: 29 616481 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(ByteBuffersDirectory@2d27e606 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@43452f28) 616481 DEBUG (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Download files to dir: [segments_1] 616481 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher tmpIndexDir_type : class org.apache.lucene.tests.store.MockDirectoryWrapper , ByteBuffersDirectory@2d27e606 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@43452f28 616482 DEBUG (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.kdm size=143 checksum=4092127899 alwaysDownload=true 616483 INFO (qtp1402283490-12959-null-11314) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11314] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.kdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616483 DEBUG (qtp1402283490-12959-null-11314) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11314] o.a.s.h.ReplicationHandler Wrote 143 bytes for file _0.kdm 616489 INFO (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 616489 ERROR (indexFetcher-8058-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher User aborted Replication => org.apache.solr.handler.IndexFetcher$ReplicationHandlerException: User aborted replication at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1826) org.apache.solr.handler.IndexFetcher$ReplicationHandlerException: User aborted replication at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1826) ~[main/:?] at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1790) ~[main/:?] at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1772) ~[main/:?] at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1192) ~[main/:?] at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:679) [main/:?] at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:416) [main/:?] at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:467) [main/:?] at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1229) [main/:?] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?] 616491 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Attempting to replicate from core [pull_replica_test_kill_pull_replica_shard1_replica_n1] on node [https://127.0.0.1:35647/solr]. 616491 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Last replication failed, so I'll force replication 616492 INFO (qtp1402283490-12965-null-11315) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11315] o.a.s.c.S.Request webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 616493 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Leader's generation: 3 616493 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Leader's version: 1728829162971 616493 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower's generation: 1 616493 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower's version: 0 616493 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Starting replication process 616494 INFO (qtp1402283490-12963-null-11316) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11316] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=0 616495 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Number of files in latest index in leader: 29 616495 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Starting download (fullCopy=true) to MockDirectoryWrapper(ByteBuffersDirectory@6127f249 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1737ddb6) 616495 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Download files to dir: [segments_1] 616495 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher tmpIndexDir_type : class org.apache.lucene.tests.store.MockDirectoryWrapper , ByteBuffersDirectory@6127f249 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1737ddb6 616496 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.kdm size=143 checksum=4092127899 alwaysDownload=true 616493 ERROR (fsyncService-8059-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.u.ExecutorUtil Uncaught exception org.apache.lucene.store.AlreadyClosedException: this Directory is closed thrown by thread: fsyncService-8059-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4 => java.lang.Exception: Submitter stack trace at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:416) java.lang.Exception: Submitter stack trace at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:416) [solr-solrj-9.8.0-SNAPSHOT.jar:9.8.0-SNAPSHOT 0456d58b8a62d412f82370dad6d2a4f2cf78933b [snapshot build, details omitted]] at org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1800) ~[main/:?] at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1772) ~[main/:?] at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1192) ~[main/:?] at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:679) ~[main/:?] at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:416) ~[main/:?] at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:467) ~[main/:?] at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$13(ReplicationHandler.java:1229) ~[main/:?] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.base/java.lang.Thread.run(Thread.java:829) [?:?] Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is closed at org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] at org.apache.lucene.store.ByteBuffersDirectory.sync(ByteBuffersDirectory.java:237) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] at org.apache.lucene.tests.store.MockDirectoryWrapper.sync(MockDirectoryWrapper.java:214) ~[lucene-test-framework-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02] at org.apache.solr.handler.IndexFetcher$DirectoryFile.sync(IndexFetcher.java:2034) ~[main/:?] at org.apache.solr.handler.IndexFetcher$FileFetcher.lambda$fetch$0(IndexFetcher.java:1803) ~[main/:?] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449) ~[solr-solrj-9.8.0-SNAPSHOT.jar:9.8.0-SNAPSHOT 0456d58b8a62d412f82370dad6d2a4f2cf78933b [snapshot build, details omitted]] ... 3 more 616497 INFO (qtp1402283490-12961-null-11317) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11317] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.kdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616499 DEBUG (qtp1402283490-12961-null-11317) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11317] o.a.s.h.ReplicationHandler Wrote 143 bytes for file _0.kdm 616499 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 143 bytes of file: _0.kdm Окт 13, 2024 8:19:23 ПА. ЧО. com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException WARNING: Uncaught exception in thread: Thread[fsyncService-8059-thread-1,5,TGRP-TestPullReplica] org.apache.lucene.store.AlreadyClosedException: this Directory is closed at __randomizedtesting.SeedInfo.seed([F30CC837FDD0DC28]:0) at org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50) at org.apache.lucene.store.ByteBuffersDirectory.sync(ByteBuffersDirectory.java:237) at org.apache.lucene.tests.store.MockDirectoryWrapper.sync(MockDirectoryWrapper.java:214) at org.apache.solr.handler.IndexFetcher$DirectoryFile.sync(IndexFetcher.java:2034) at org.apache.solr.handler.IndexFetcher$FileFetcher.lambda$fetch$0(IndexFetcher.java:1803) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 616506 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.si size=473 checksum=1097052905 alwaysDownload=true 616506 INFO (qtp1402283490-12957-null-11318) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11318] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.si&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616509 DEBUG (qtp1402283490-12957-null-11318) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11318] o.a.s.h.ReplicationHandler Wrote 473 bytes for file _0.si 616509 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 473 bytes of file: _0.si 616510 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.kdi size=68 checksum=363169705 alwaysDownload=true 616510 INFO (qtp1402283490-12967-null-11319) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11319] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.kdi&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616510 DEBUG (qtp1402283490-12967-null-11319) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11319] o.a.s.h.ReplicationHandler Wrote 68 bytes for file _0.kdi 616511 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 68 bytes of file: _0.kdi 616511 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.fdm size=157 checksum=841077555 alwaysDownload=true 616512 INFO (qtp1402283490-12959-null-11320) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11320] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616512 DEBUG (qtp1402283490-12959-null-11320) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11320] o.a.s.h.ReplicationHandler Wrote 157 bytes for file _0.fdm 616513 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 157 bytes of file: _0.fdm 616513 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.doc size=77 checksum=985266366 alwaysDownload=true 616513 INFO (qtp1402283490-12965-null-11321) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11321] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_Lucene99_0.doc&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616514 DEBUG (qtp1402283490-12965-null-11321) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11321] o.a.s.h.ReplicationHandler Wrote 77 bytes for file _0_Lucene99_0.doc 616514 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 77 bytes of file: _0_Lucene99_0.doc 616514 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.tim size=102 checksum=1906253852 alwaysDownload=true 616515 INFO (qtp1402283490-12963-null-11322) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11322] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_Lucene99_0.tim&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616515 DEBUG (qtp1402283490-12963-null-11322) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11322] o.a.s.h.ReplicationHandler Wrote 102 bytes for file _0_Lucene99_0.tim 616516 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 102 bytes of file: _0_Lucene99_0.tim 616516 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.fdx size=64 checksum=4008489842 alwaysDownload=true 616516 INFO (qtp1402283490-12961-null-11323) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11323] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616517 DEBUG (qtp1402283490-12961-null-11323) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11323] o.a.s.h.ReplicationHandler Wrote 64 bytes for file _0.fdx 616517 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 64 bytes of file: _0.fdx 616517 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.tip size=74 checksum=204394137 alwaysDownload=true 616518 INFO (qtp1402283490-12957-null-11324) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11324] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_Lucene99_0.tip&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616518 DEBUG (qtp1402283490-12957-null-11324) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11324] o.a.s.h.ReplicationHandler Wrote 74 bytes for file _0_Lucene99_0.tip 616519 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 74 bytes of file: _0_Lucene99_0.tip 616519 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.fdt size=114 checksum=2080799656 alwaysDownload=true 616520 INFO (qtp1402283490-12967-null-11325) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11325] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616520 DEBUG (qtp1402283490-12967-null-11325) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11325] o.a.s.h.ReplicationHandler Wrote 114 bytes for file _0.fdt 616521 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 114 bytes of file: _0.fdt 616521 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0_Lucene90_0.dvm size=758 checksum=3872417709 alwaysDownload=true 616522 INFO (qtp1402283490-12959-null-11326) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11326] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_Lucene90_0.dvm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616522 DEBUG (qtp1402283490-12959-null-11326) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11326] o.a.s.h.ReplicationHandler Wrote 758 bytes for file _0_Lucene90_0.dvm 616523 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 758 bytes of file: _0_Lucene90_0.dvm 616523 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0_Lucene90_0.dvd size=81 checksum=3626414175 alwaysDownload=true 616523 INFO (qtp1402283490-12965-null-11327) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11327] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_Lucene90_0.dvd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616524 DEBUG (qtp1402283490-12965-null-11327) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11327] o.a.s.h.ReplicationHandler Wrote 81 bytes for file _0_Lucene90_0.dvd 616527 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 81 bytes of file: _0_Lucene90_0.dvd 616529 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.kdd size=79 checksum=3053912650 alwaysDownload=true 616530 INFO (qtp1402283490-12963-null-11328) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11328] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.kdd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616530 DEBUG (qtp1402283490-12963-null-11328) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11328] o.a.s.h.ReplicationHandler Wrote 79 bytes for file _0.kdd 616531 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 79 bytes of file: _0.kdd 616531 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0_Lucene99_0.tmd size=252 checksum=95871406 alwaysDownload=true 616532 INFO (qtp1402283490-12961-null-11329) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11329] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_Lucene99_0.tmd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616532 DEBUG (qtp1402283490-12961-null-11329) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11329] o.a.s.h.ReplicationHandler Wrote 252 bytes for file _0_Lucene99_0.tmd 616532 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 252 bytes of file: _0_Lucene99_0.tmd 616533 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_0.fnm size=660 checksum=2938713832 alwaysDownload=true 616533 INFO (qtp1402283490-12957-null-11330) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11330] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616534 DEBUG (qtp1402283490-12957-null-11330) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11330] o.a.s.h.ReplicationHandler Wrote 660 bytes for file _0.fnm 616534 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 660 bytes of file: _0.fnm 616534 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.si size=473 checksum=3349947083 alwaysDownload=true 616535 INFO (qtp1402283490-12967-null-11331) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11331] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.si&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616535 DEBUG (qtp1402283490-12967-null-11331) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11331] o.a.s.h.ReplicationHandler Wrote 473 bytes for file _1.si 616536 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 473 bytes of file: _1.si 616536 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1_Lucene90_0.dvd size=81 checksum=1932654087 alwaysDownload=true 616537 INFO (qtp1402283490-12959-null-11332) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11332] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_Lucene90_0.dvd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616537 DEBUG (qtp1402283490-12959-null-11332) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11332] o.a.s.h.ReplicationHandler Wrote 81 bytes for file _1_Lucene90_0.dvd 616537 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 81 bytes of file: _1_Lucene90_0.dvd 616538 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1_Lucene99_0.tmd size=252 checksum=3610015843 alwaysDownload=true 616539 INFO (qtp1402283490-12965-null-11333) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11333] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_Lucene99_0.tmd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616539 DEBUG (qtp1402283490-12965-null-11333) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11333] o.a.s.h.ReplicationHandler Wrote 252 bytes for file _1_Lucene99_0.tmd 616540 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 252 bytes of file: _1_Lucene99_0.tmd 616540 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.kdd size=79 checksum=3998912920 alwaysDownload=true 616541 INFO (qtp1402283490-12963-null-11334) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11334] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.kdd&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616541 DEBUG (qtp1402283490-12963-null-11334) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11334] o.a.s.h.ReplicationHandler Wrote 79 bytes for file _1.kdd 616542 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 79 bytes of file: _1.kdd 616542 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.fnm size=660 checksum=3214671176 alwaysDownload=true 616542 INFO (qtp1402283490-12961-null-11335) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11335] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fnm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616543 DEBUG (qtp1402283490-12961-null-11335) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11335] o.a.s.h.ReplicationHandler Wrote 660 bytes for file _1.fnm 616543 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 660 bytes of file: _1.fnm 616543 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.kdi size=68 checksum=532616558 alwaysDownload=true 616545 INFO (qtp1402283490-12957-null-11336) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11336] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.kdi&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616545 DEBUG (qtp1402283490-12957-null-11336) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11336] o.a.s.h.ReplicationHandler Wrote 68 bytes for file _1.kdi 616546 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 68 bytes of file: _1.kdi 616546 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.fdm size=157 checksum=855728491 alwaysDownload=true 616547 INFO (qtp1402283490-12967-null-11337) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11337] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616547 DEBUG (qtp1402283490-12967-null-11337) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11337] o.a.s.h.ReplicationHandler Wrote 157 bytes for file _1.fdm 616547 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 157 bytes of file: _1.fdm 616547 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1_Lucene90_0.dvm size=758 checksum=2429945466 alwaysDownload=true 616548 INFO (qtp1402283490-12959-null-11338) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11338] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_Lucene90_0.dvm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616549 DEBUG (qtp1402283490-12959-null-11338) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11338] o.a.s.h.ReplicationHandler Wrote 758 bytes for file _1_Lucene90_0.dvm 616549 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 758 bytes of file: _1_Lucene90_0.dvm 616549 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1_Lucene99_0.tip size=74 checksum=214641708 alwaysDownload=true 616550 INFO (qtp1402283490-12965-null-11339) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11339] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_Lucene99_0.tip&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616550 DEBUG (qtp1402283490-12965-null-11339) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11339] o.a.s.h.ReplicationHandler Wrote 74 bytes for file _1_Lucene99_0.tip 616551 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 74 bytes of file: _1_Lucene99_0.tip 616551 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.fdx size=64 checksum=4107293692 alwaysDownload=true 616552 INFO (qtp1402283490-12963-null-11340) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11340] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdx&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616552 DEBUG (qtp1402283490-12963-null-11340) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11340] o.a.s.h.ReplicationHandler Wrote 64 bytes for file _1.fdx 616553 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 64 bytes of file: _1.fdx 616553 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.kdm size=143 checksum=798627444 alwaysDownload=true 616554 INFO (qtp1402283490-12961-null-11341) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11341] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.kdm&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616554 DEBUG (qtp1402283490-12961-null-11341) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11341] o.a.s.h.ReplicationHandler Wrote 143 bytes for file _1.kdm 616554 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 143 bytes of file: _1.kdm 616555 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1_Lucene99_0.doc size=77 checksum=3918513493 alwaysDownload=true 616555 INFO (qtp1402283490-12957-null-11342) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11342] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_Lucene99_0.doc&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616556 DEBUG (qtp1402283490-12957-null-11342) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11342] o.a.s.h.ReplicationHandler Wrote 77 bytes for file _1_Lucene99_0.doc 616556 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 77 bytes of file: _1_Lucene99_0.doc 616556 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1_Lucene99_0.tim size=102 checksum=2440597014 alwaysDownload=true 616557 INFO (qtp1402283490-12967-null-11343) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11343] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_Lucene99_0.tim&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616557 DEBUG (qtp1402283490-12967-null-11343) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11343] o.a.s.h.ReplicationHandler Wrote 102 bytes for file _1_Lucene99_0.tim 616558 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 102 bytes of file: _1_Lucene99_0.tim 616558 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=_1.fdt size=114 checksum=808921209 alwaysDownload=true 616559 INFO (qtp1402283490-12959-null-11344) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11344] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdt&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616559 DEBUG (qtp1402283490-12959-null-11344) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11344] o.a.s.h.ReplicationHandler Wrote 114 bytes for file _1.fdt 616560 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 114 bytes of file: _1.fdt 616560 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Downloading file=segments_3 size=302 checksum=2730871434 alwaysDownload=true 616560 INFO (qtp1402283490-12965-null-11345) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11345] o.a.s.c.S.Request webapp=/solr path=/replication params={generation=3&qt=/replication&file=segments_3&checksum=true&wt=filestream&version=2.2&command=filecontent} status=0 QTime=0 616561 DEBUG (qtp1402283490-12965-null-11345) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11345] o.a.s.h.ReplicationHandler Wrote 302 bytes for file segments_3 616561 DEBUG (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Fetched and wrote 302 bytes of file: segments_3 616562 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Bytes downloaded: 6506, Bytes skipped downloading: 0 616562 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=true,bytesDownloaded=6506) : 0 secs (null bytes/sec) to MockDirectoryWrapper(ByteBuffersDirectory@6127f249 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1737ddb6) 616562 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.SolrCore Updating index properties... index=index.20241013201923273 616562 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher removing old index directory MockDirectoryWrapper(ByteBuffersDirectory@1c24eec4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@660d8cf4) 616563 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. 616564 INFO (searcherExecutor-8053-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms 616565 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Replication Recovery was successful. 616565 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Restarting background replicate from leader process 616565 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.ZkController pull_replica_test_kill_pull_replica_shard1_replica_p2 starting background replication from leader 616565 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:01 616566 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms 616566 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms 616566 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Registering as Active after recovery. 616567 DEBUG (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.ReplicationHandler Polling for index modifications 616567 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Finished recovery process, successful=[true] msTimeTaken=113.0 616567 INFO (zkCallback-8044-thread-2) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 218] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616567 INFO (recoveryExecutor-8042-thread-1-processing-127.0.0.1:38207_solr pull_replica_test_kill_pull_replica_shard1_replica_p2 pull_replica_test_kill_pull_replica shard1 core_node4) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.c.RecoveryStrategy Finished recovery process. recoveringAfterStartup=true msTimeTaken=113.0 616567 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 218] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616567 INFO (zkCallback-8007-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 218] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616568 INFO (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Updated leaderUrl to https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/ 616569 INFO (qtp1402283490-12963-null-11346) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11346] o.a.s.c.S.Request webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 616569 INFO (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Leader's generation: 3 616569 INFO (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Leader's version: 1728829162971 616569 INFO (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower's generation: 3 616569 INFO (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower's version: 1728829162971 616569 INFO (indexFetcher-8062-thread-1) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:] o.a.s.h.IndexFetcher Follower in sync with leader. 616575 INFO (qtp1402283490-12967-null-11347) [n:127.0.0.1:35647_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node3 x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11347] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11347 hits=2 status=0 QTime=0 616575 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node3 (https://127.0.0.1:35647/solr/pull_replica_test_kill_pull_replica_shard1_replica_n1/) has all 2 docs 616606 INFO (qtp1091538342-13057-null-11348) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11348] o.a.s.c.S.Request webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} rid=null-11348 hits=2 status=0 QTime=0 616607 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Replica core_node4 (https://127.0.0.1:38207/solr/pull_replica_test_kill_pull_replica_shard1_replica_p2/) has all 2 docs 616608 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica tearDown deleting collection 616609 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.h.i.e.RetryExec I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {s}->https://127.0.0.1:38207: The target server failed to respond 616609 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.h.i.e.RetryExec Retrying request to {s}->https://127.0.0.1:38207 616623 INFO (OverseerThreadFactory-7998-thread-2) [n: c:pull_replica_test_kill_pull_replica s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null 616624 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11350] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_kill_pull_replica.shard1.replica_n1 tag=null 616624 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11350] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@28cf8c44 pull_replica_test_kill_pull_replica_shard1_replica_n1 616624 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11350] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_kill_pull_replica.shard1.replica_n1 tag=SolrCore@28cf8c44 616624 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11350] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.pull_replica_test_kill_pull_replica.shard1.leader tag=SolrCore@28cf8c44 616625 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_n1 t:null-11350] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 616632 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x: t:null-11350] o.a.s.c.ZkShardTerms Successful update of terms at /collections/pull_replica_test_kill_pull_replica/terms/shard1 to Terms{values={}, version=2} for removeTerm 616633 INFO (zkCallback-8044-thread-2) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 224] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616634 INFO (zkCallback-7988-thread-1) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 224] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616634 INFO (qtp1402283490-12961-null-11350) [n:127.0.0.1:35647_solr c: s: r: x: t:null-11350] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=pull_replica_test_kill_pull_replica_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=9 616646 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11350] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2 tag=null 616646 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11350] o.a.s.c.ZkController pull_replica_test_kill_pull_replica_shard1_replica_p2 stopping background replication from leader 616646 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11350] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@3702f0db pull_replica_test_kill_pull_replica_shard1_replica_p2 616647 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11350] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_kill_pull_replica.shard1.replica_p2 tag=SolrCore@3702f0db 616647 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11350] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.pull_replica_test_kill_pull_replica.shard1.leader tag=SolrCore@3702f0db 616647 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x:pull_replica_test_kill_pull_replica_shard1_replica_p2 t:null-11350] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close() ... SKIPPED (unnecessary). 616649 INFO (qtp1091538342-13055-null-11350) [n:127.0.0.1:38207_solr c: s: r: x: t:null-11350] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=pull_replica_test_kill_pull_replica_shard1_replica_p2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=3 616649 INFO (zkCallback-8044-thread-2) [n: c: s: r: x: t:] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_kill_pull_replica/state.json zxid: 225] for collection [pull_replica_test_kill_pull_replica] has occurred - updating... (live nodes size: [2]) 616652 INFO (OverseerThreadFactory-7998-thread-2) [n: c:pull_replica_test_kill_pull_replica s: r: x: t:] o.a.s.c.c.ZkStateReader /collections/pull_replica_test_kill_pull_replica/state.json is deleted, stop watching children 616660 INFO (qtp1091538342-13058-null-11349) [n:127.0.0.1:38207_solr c:pull_replica_test_kill_pull_replica s: r: x: t:null-11349] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=pull_replica_test_kill_pull_replica&action=DELETE&wt=javabin&version=2} status=0 QTime=46 616661 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.c.TestPullReplica Collection deleted 616661 INFO (TEST-TestPullReplica.testKillPullReplica-seed#[F30CC837FDD0DC28]) [n: c: s: r: x: t:] o.a.s.SolrTestCaseJ4 ###Ending testKillPullReplica